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Single-Step Conversion from RGB 
Bayer Pattern to YUV 4:2:0 

Format 

Background of Invention 

[0001] This invention relates to digital imaging, and more particularly to conversion from 
RGB to YUV formats. 

H 

O [0002] Digital imaging is widely deployed in stand-alone digital cameras and camcorders. 

Q 

S Other devices such as cell phones and personal digital assistants (PDA's) sometimes 

4 s include digital imaging components, especially as digital imaging is improved and 

Si lowered in cost. Images from digital cameras can be downloaded and stored on 
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personal computers or wirelessly transmitted after capture from an integrated device. 
Video as well as still images can be captured, depending on the kind of digital camera. 

[0003] Figure 1 is a block diagram for a typical digital camera. Light focused through a 
lens is directed toward sensor 1 2, which can be a charge-coupled device (CCD) array 
or a complementary metal-oxide-semiconductor (CMOS) sensor array. The light 
falling on the array generates electrical currents, which are amplified by analog amp 
1 4 before being converted from analog to digital values by A/D converter 1 6. An 8, 9, 
or 1 0-bit mono-color pixel is output to interpolator 1 0. These mono-color pixels are 
in a Bayer-pattern as shown in Figure 2. Each pixel is either a red, a blue, or a green 
intensity. Each pixel has only one of the 3 color components rather than all 3 
components per pixel. 



[0004] 



The R, G, or B digital values in the Bayer (or Beyer) pattern are processed by 
interpolator 10 to interpolate missing pixels in the Bayer pattern. The filled-in pixels 
each have all 3 color components - R, G, and B. The filled-in RGB pixels can then be 
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displayed on display 19. 

[0005] Processor 1 5 converts the filled-in pattern of RGB pixels from interpolator 10 to 
luminance-chrominance YUV pixels. YUV pixels often have a 4:4:4 format, with 8 bits 
for each of 2 colors and for the luminance. However, other YUV formats such as 4:2:0 
may be required by compressor 1 8, so second converter 1 1 may be required to 
convert 4:4:4 YUV pixels to 4:2:2 format, while third converter 1 1 ' converts 4:2:2 YUV 
pixels to 4:2:0 format. The converted YUV pixels can then be compressed by 
compressor 1 8 and stored on disk 1 7 or on a solid-state memory. 

[0006] Sensor 1 2 detects red, blue and green colors. However, each array point in sensor 
12 can detect only one of the three primary colors. Rather than outputting an RGB 
pixel, sensor 1 2 can output only a single-color pixel at any given time. For example, a 
line of pixels output by sensor 1 2 might have a red pixel followed by a green pixel. 
Another line might have alternating green and blue pixels. 

[0007] Each pixel represents the intensity of one of the primary colors at a point in the 
sensor array. Thus a red pixel indicates the intensity of red light at a point, while a 
neighboring green pixel indicates the intensity of green light at the next point in the 
sensor array. Each pixel thus contains only one-third of the total color information. 

[0008] The remaining color information is obtained by interpolation. The green intensity 
of a red pixel is calculated by averaging the green intensities of neighboring green 
pixels. The blue intensity for that red pixel is calculated by averaging or interpolating 
the nearest blue pixels. Interpolator 10 performs this color interpolation, calculating 
the missing primary-color intensities for each pixel location. 

[0009] The electrical currents produced by the different primary colors can vary, 

depending on the sensor used and the wavelength and energy of the light photons. 
An adjustment known as a white-balance is often performed before interpolator 10, 
either on analog or digital values. Each primary color can be multiplied by a different 
gain to better balance the colors. Compensation can also be made for different 
lighting conditions, increasing all primary colors for dark pictures or decreasing all 
colors for bright pictures (overexposure). 

[001 0] Bayer Pattern Fig. 2 
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[001 1] Figure 2 shows an image captured by a sensor that generates single-color pixels 
in a Bayer pattern. The example shows an 800 x 600 frame or image for display in the 
common super-VGA resolution. A total of 600 lines are captured by the sensor, with 
800 pixels per line. 

[001 2] Personal computers and many other devices display full-color pixels that have all 
three primary-color intensities (RGB). In contrast, the sensor in a digital camera can 
detect only one of the three primary colors for each point in the 800 x 600 sensor 
array. Detectors for green are alternated with red detectors in the first line, while 
green detectors are alternated with blue detectors in the second line. 

[001 3] The first horizontal line and each odd line have alternating red and green 

detectors, so pixels output from these odd lines are in a G-R-G-R-G-R-G sequence. 
The second horizontal line and each even line have alternating green and blue 
detectors, so pixels output from these even lines are in a B-G-B-G-B-C-B sequence. 

[001 4] Half of the pixels are green pixels, while one-quarter of the pixels are read and 

the last quarter are blue. The green pixels form a checkerboard pattern, with blue and 
red pixels surrounded by green pixels. Since the human eye is more sensitive to 
green, the Bayer pattern has more green pixels than red or blue. 

[001 5] The green intensity for a red pixel location can be interpolated by averaging the 
four green pixels that surround the red pixel. For example, the green intensity for red 
pixel at location (3,2) is the sum of green pixels (3,1), (3,3), (2,2), and (4,2), divided 
by four. Likewise, the green intensity for a blue pixel location can be interpolated by 
averaging the four surrounding green pixels. For blue pixel (2,3), the interpolated 
green intensity is the sum of green pixels (2,2), (2,4), (1,3), and (3,3), divided by four. 

[001 6] The red and blue values for a green pixel location can also be calculated from the 
2 red and 2 blue pixels that surround each green pixel. For green pixel (2,2), the 
interpolated red value is the average of red pixels (1 ,2) and (3,2) above and below the 
green pixel, while the interpolated blue value is the average of blue pixels (2,1) and 
(2,3) to the right and left of the green pixel. 

[0017] 

While such digital-camera processors are useful, cost reduction is desirable since 
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digital cameras are price-sensitive consumer devices and may be a component of an 
integrated device such as a cell phone or PDA. Such integrated devices may use image 
compression to allow pictures to be compactly transmitted over wireless networks. 
Compression may require one or more format conversions. These format conversions 
add to complexity and cost of the device. 

[001 8] Some devices may use displays that use YUV format rather than RGB. Devices 

could benefit from reduced cost and complexity if the number of format conversion 
could be reduced. However, since the Bayer pattern has so many missing pixels, some 
type of interpolation step seems necessary. 

[001 9] What is desired is a digital-image processor that does not require interpolation on 
RGB pixels in the Bayer pattern. It is desired to combine color interpolation of Bayer- 
pattern pixels with format conversion to YUV. It is desired to use a single step 
conversion that both interpolates and generates YUV formats from a Bayer-pattern 
input. 



+■ Brief Description of Drawings 

H 
M 

[0020] Figure 1 is a block diagram for a typical digital camera. 
Q [0021] Figure 2 shows an image captured by a sensor that generates single-color pixels 

01 

1^ in a Bayer pattern. 

M 

O [0022] Figure 3 highlights a direct conversion from RGB Bayer pattern to YUV. 

nJ 

[0023] Figure 4 shows a direct-conversion image processor using a 5-line Bayer-pattern 
buffer and a 3-line luminance buffer. 

[0024] Figure 5 illustrates locations of generated Y, U, and V components for five rows of 
Bayer-pattern pixels stored in the 5-line buffer. 

[0025] Figure 6 highlights luminance calculation from a Bayer-pattern input without 
intermediate RGB interpolation. 

[0026] Figures 7A-C illustrate chrominance calculation from Bayer-pattern input pixels 
and luminance pixels without intermediate RGB interpolation. 
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[0027] Figures 8A-D illustrate patterns matched for luminance calculation and 

coefficients for generating luminance pixels without intermediate RGB interpolation. 

[0028] Figures 9A-D illustrate the pattern matched for chrominance calculation and 
coefficients for generating intermediate sums without RGB interpolation. 

[0029] Figure 1 0 shows using an associative array processor to directly generate YUV 
pixels from an RGB Bayer-pattern without intermediate interpolation. 

[0030] Figure 1 1 shows using a digital-signal processor (DSP) to perform luminance and 
chrominance calculations. 

[0031] Figures 1 2A-D illustrate for an alternate Bayer pattern initialization the patterns 
matched for chrominance calculation and coefficients for generating intermediate 
sums without RGB interpolation. 

Detailed Description 

[0032] The present invention relates to an improvement in digital imaging. The following 
description is presented to enable one of ordinary skill in the art to make and use the 
invention as provided in the context of a particular application and its requirements. 
Various modifications to the preferred embodiment will be apparent to those with skill 
in the art, and the general principles defined herein may be applied to other 
embodiments. Therefore, the present invention is not intended to be limited to the 
particular embodiments shown and described, but is to be accorded the widest scope 
consistent with the principles and novel features herein disclosed. 

[0033] While interpolation could be performed on RGB pixels of the Bayer pattern, the 
inventors have realized that interpolation can be included in a single re-formatting 
step that converts RGB pixels to YUV pixels. Using such an integrated direct 
conversion step can reduce complexity and cost, as the intermediate full-pattern of 
interpolated RGB pixels are not generated. 

[0034] Figure 3 highlights a direct conversion from RGB Bayer pattern to YUV. Light 
captured by sensor front-end 22 is converted to electrical signals by a charge- 
coupled or transistor device, and the electrical signals are converted from analog to 
digital format. 
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[0035] The image sensor does not generate all three primary color components (RGB) for 
each pixel. Instead only one of the three color-components is produced per pixel 
location. The colors produced alternate from pixel to pixel, and from line to line in a 
Bayer pattern as shown in Figure 2. Typically one pixel with only one of the three 
primary colors is output for each clock cycle. 

[0036] Direct converter 28 receives the Bayer-pattern input from front-end 22. The input 
Bayer pattern has missing color components. Direct converter 28 uses digital filters to 
both interpolate the missing color components and convert from RGB to YUV format. 
In particular, the pixels are directly converted to the YUV 4:2:0 format, and an 
intermediate YUV 4:4:4 format is not needed. 

[0037] The output of direct converter 28 includes brightness or luminance array 36 of Y 
pixels. One Y pixel is output for each pixel location in Bayer pattern 32 produced by 
S front-end 22. Luminance array 36 thus has the same width W and height H as the 

O original Bayer pattern 32. 

yl 

£ 

[0038] Direct converter 28 also outputs chrominance U, V arrays 38, 39 that contain color 
% f information. The YUV 4:2:0 format generates one-quarter as many U pixels as Y 

s pixels, so U array 38 has a width of W/2 and a height of H/2. Likewise, only one in 

four pixels of original Bayer pattern 32 has a V pixel generated by direct converter 28, 
M so V array 39 has height H/2 and width W/2. 

[0039] Both interpolation and format conversion are performed in one discrete step using 
direct converter 28. Intermediate arrays of pixel data, such as for filled-in RGB and 
YUV 4:4:4 pixels are not necessary, although circuitry could be added to generate 
these, such as when an RGB display is used. 



[0040] 



Figure 4 shows a direct-conversion image processor using a 5-line Bayer-pattern 
buffer and a 3-line luminance buffer. Light captured by sensor front-end 22 is 
converted to electrical signals by a charge-coupled, transistor, or other sensing 
device, and the electrical signals are converted from analog to digital format. The 
image sensor does not generate all three primary color components (RGB) for each 
pixel, but only produces one of the three color-components per pixel. The colors 
produced alternate from pixel to pixel, and from line to line in a Bayer pattern as 
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[0046] 



shown in Figure 2. Typically one pixel with only one of the three primary colors is 
output for each clock cycle. 

The sensitivity of the sensor to the different primary colors is not equal. Some 
colors experience more attenuation than others. Also, the image may be under or over 
exposed. White balancer 26 multiplies each pixel from front end 22 by a gain. Red 
pixels are multiplied by a red gain, blue pixels by a blue gain, and green pixels by a 
green gain. The pixel-gain product is output from white balancer 26 to line buffer 30. 

The pixel gains may be pre-set gains, or may be calculated on the fly. Pre-set 
gains can be calculated at the factory for the sensor type and fixed gains stored in a 
register and applied to white balancer 26. Video digital cameras produce a series of 
frames, and still cameras are normally pointed toward the picture before shooting, so 
gain calculator 24 can generate a running average of all pixels for each of the 3 colors 
from one or more frames, and use the averages to calculate gains for the next frame. 
Averages can also be generated for just a portion of an image frame, and the gains 
applied to the entire image or a portion of the image. 

Five-line buffer 30 contains about five lines of pixels, rather than all 600 or more 
lines. For SVGA resolution, each line contains 800 pixels, so line buffer 30 contains 
800 x 5 or 4,000 pixels. Other resolutions, such as 1 024x768 have more pixels per 
line, so the size of line buffer 30 can increase. 

Luminance calculator 40 receives the R, G, and B Bayer-pattern pixels from buffer 
30 and generates a luminance (Y) value for each pixel location. Rather than simply 
convert each single Bayer-pattern pixel to a Y pixel, an averaging filter is used that 
receives 9 pixels from 3 lines of the Bayer pattern in buffer 30. Using 9 pixels rather 
than just 1 allows for more colors and locations to be included when calculating the Y 
pixel at the center of the 9-pixel block or box. 

Since each Y pixel is generated from 3 lines of Bayer-pattern pixels, the 5 lines of 
buffer 30 are reduced to 3 lines of Y pixels that are stored in three-line Y buffer 34. 

Luminance calculator 40 performs both horizontal and vertical interpolation over 
the 9-pixel-box input to generate the Y pixel in the center of the 3x3 box. All three 
color components (R,G,B) are included in the 3x3 input since the R, C, and B pixels 
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alternate in the Bayer pattern. Only the center pixel and neighboring Bayer-pattern 
pixels that touch the center pixel's location at the center of the 3x3 box are input to 
luminance calculator 40. There are a total of 8 such neighboring pixels three pixels on 
three lines. 

[0047] The weighting or filter coefficients for each pixel in the 3x3 box varies by color, 
since there are twice as many green pixels in a Bayer pattern as there are red or blue 
pixels. The green pixels form a checkerboard pattern with green pixels on every line, 
but the red or blue pixels form a more open pattern with pixels only on alternating 
lines. Thus different filter coefficients are applied to pixels depending on the current 
location within the Bayer pattern. 

[0048] Chrominance calculator 42 generates either a U or a V chrominance pixel value for 
each block of four pixel locations of the Bayer pattern. Blue (B) pixels within the 3x3 
box from the Bayer pattern are input from buffer 30 to chrominance calculator 42 
when the U component is calculated, while red (R) pixels within the 3x3 Bayer-pattern 
box are input to chrominance calculator 42 from buffer 30 when the V component is 
calculated. 

[0049] In addition to the R or B Bayer-pattern pixels input from buffer 30, a 3x3 box of 
luminance (Y) values are input from 3-line buffer 34 to chrominance calculator 42. An 
average of the 9 Y values is included when generating the chrominance value. Thus 
chrominance calculator 42 receives red (R) and blue (B) Bayer-pattern pixels, and Y 
luminance values, but does not receive green (G) pixels from the Bayer pattern. 

[0050] A single line of U and V components are generated by chrominance calculator 42 
from the 5 lines in buffer 30. The line of U and V components can be stored to U and 
V arrays for later packing into the YUV format. Three lines of Y components are 
generated by luminance calculator 40 from the 5 lines in buffer 30. One Y line is 
generated at a time, but a total of 3 lines are stored in buffer 34. The Y values from 
luminance calculator 40 can likewise be stored in an array or packed into the desired 
YUV format and output as a bitstream. 

[0051] 

Figure 5 illustrates locations of generated Y,U, and V components for five rows of 
Bayer-pattern pixels stored in the 5-line buffer. The R, C, and B pixels of the Bayer- 
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pattern input are stored in five-line buffer 30 for later input to the luminance and 
chrominance calculators. Each pixel in the diagram has a row and column number 
indicating its location. For example, the second pixel on the fourth row is G42, 
indicating a green pixel. There is no red or blue pixel for location 4,2 since the Bayer 
pattern is not filled in. 

[0052] Nine Bayer-pattern pixels at a time are input to the luminance calculator. When 
luminance pixel Y32 is to be generated, filter 94 inputs nine pixels: 

[0053] B21 , G22, B23 from row 2, 

[0054] G31 , R32, G33 from row 3, and 

[0055] B41 , G42, B43 from row 4. 

[0056] This 3x3 box has a red pixel in the center position, and matches pattern 3 of Fig. 
8C. The filter coefficients of Fig. 8C are then multiplied by the corresponding Bayer- 
pattern pixels from input filter 94 and the nine products summed to generate the Y 
pixel for the center of the 3x3 box, pixel Y32. 

[0057] The U and V chrominance pixels are not generated for location 3,2, due to down- 
sampling. 

[0058] For the following pixel location (3,3), the input location is moved filter 96. Filter 
96 input the nine pixels: 

[0059] G22, B23, G24 from row 2, 

[0060] R32, G33, R34 from row 3, and 

[0061] G42, B43, G44 from row 4. 

[0062] This 3x3 box has a green pixel in the center position, and red pixels to the right 
and left of the center pixel, and matches pattern 4 of Fig. 8D. The filter coefficients of 
Fig. 8D are then multiplied by the corresponding Bayer-pattern pixels from input filter 
96 and the nine products summed to generate the Y pixel for the center of the 3x3 
box, pixel Y33. 

[0063] The U and V chrominance pixels are generated for location 3,3 by the 
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chrominance calculator using input filter 96. Only the two blue pixels B23 and B43 are 
input to the chrominance calculator to generate U33. The other red and green pixels 
are ignored. Likewise, only the two red pixels R32 and R34 are input to the 
chrominance calculator to generate V33 while the other blue and green pixels are 
ignored. However, all nine luminance Y pixels generated for locations in the 3x3 box 
of filter 96 are also input to chrominance calculator and used to calculate U33 and 
V33. 

The U and V chrominance pixels are not generated for locations 3,2 and 3,4. Also, 
the U, V components are not generated for alternating lines, such as for lines 2 and 4. 
The YUV 4:2:0 format has half as many U and V pixels as Y pixels, so every other pixel 
location skips generation of U and V components and alternate lines generate no U or 
V components. Also, the bit-size of the U and V pixels is half that of the Y pixels. 
Thus there is a higher luminance precision than for each chrominance component, but 
the overall chrominance precision matches the luminance precision using this format. 

Figure 6 highlights luminance calculation from a Bayer-pattern input without 
intermediate RGB interpolation. Three columns of 5-line buffer 30 are shown. Bayer- 
pattern pixels G22, B23, G24, R32, G33, R34, G42, B43, G44 are input to calculator 
46, which multiplies each of the 9 input pixels by a corresponding coefficient, and 
then sums the 9 products to generate the luminance at the center of the 3x3 box. 
Pixel Y33 is output to three-line buffer 34 and is also stored in the output Y array. 

The upper and lower Y values Y23, Y43 may be calculated in parallel with Y33, or 
may be generated at different times. Upper calculator 44 receives 9 Bayer-pattern 
pixels Rl 2, Gl 3, Rl 4, G22, B23, G24, R32, G33, R34. Upper calculator 44 multiplies 
each of the 9 input pixels by a corresponding coefficient, and sums the 9 products to 
generate the luminance at the center of the upper 3x3 box. Pixel Y23 is output to 
three-line buffer 34 and is also stored in the output Y array. 

[0067] Similarly, lower calculator 48 receives 9 Bayer-pattern pixels R32, G33, R34, G42, 
B43, G44, R52, G53, R54. Lower calculator 48 multiplies each of its 9 input pixels by a 
corresponding coefficient, and sums the 9 products to generate the luminance at the 
center of the lower 3x3 box. Pixel Y43 is output to three-line buffer 34 and is also 
stored in the output Y array. 
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[0068] Luminance calculator 40 can include calculators 44, 46, 48 operating in parallel, 
or one calculator could be used for each calculation in sequence, but with the input 
filter and output locations shifted. 

[0069] Figures 7A-C illustrate chrominance calculation from Bayer-pattern input pixels 
and luminance pixels without intermediate RGB interpolation. A 3x3 box of luminance 
(Y) pixels are read from 3-line buffer 34. In this example, center location 3,3 is being 
calculated, so luminance pixels Y22, Y23, Y24, Y32, Y33, Y34, Y42, Y43, Y44 are 
input to calculator 50 in chrominance calculator 42. Calculator 50 generates an 
average luminance of the 9 input pixels by multiplying each of the nine Y pixels by a 
coefficient equal to 1 /9, and then summing the 9 products. 

[0070] The result from calculator 50, labeled "Y/9" is temporarily stored and used to 
calculate both the U and V components for the center pixel, U33, V33. The 
intermediate Y/9 luminance average can be stored in a temporary or pipeline buffer 
(not shown) other than chrominance output array 58. 

[0071] Figure 7B highlights calculation of the U chrominance component from the 

luminance average and blue pixels of the un-interpolated Bayer-pattern. The Y/9 
luminance average from calculator 50 is input to calculator 52 of chrominance 
calculator 42. Only the blue pixels within the 3x3 input box are input to calculator 52. 
The green and red pixels in the 3x3 box are ignored. Since only 2 of the 9 pixels in 
the 3x3 box are blue pixels, only two Bayer-pattern pixels are input to calculator 52. 
The two blue pixels input are B23 and B43. 

[0072] Calculator 52 multiplies the two blue pixels by corresponding coefficients shown 
in Fig. 9C, and sums the two products to get an average blue pixel (S_U). The Y/9 
luminance average is then subtracted from the average blue pixel, and the difference 
is multiplied by a constant to generate the U component at the center of the 3x3 box, 
U33. The generated U component can then be stored in chrominance output array 58 
or further operated upon for compression or packing. 



[0073] 



Figure 7C highlights calculation of the V chrominance component from the 
luminance average and red pixels of the un-interpolated Bayer- pattern. The Y/9 
luminance average from calculator 50 is input to calculator 54 of chrominance 



APP_ID=10064177 



Page 11 of 37 



calculator 42. Only the red pixels within the 3x3 input box are input to calculator 54. 
The green and blue pixels in the 3x3 box are ignored. Since only 2 of the 9 pixels in 
the 3x3 box are red pixels, only two Bayer-pattern pixels are input to calculator 54. 
The two red pixels input are R32 and R34. 

[0074] Calculator 54 multiplies the two red pixels by corresponding coefficients shown in 
Fig. 9D, and sums the two products to get an average red pixel (S_V). The Y/9 
luminance average is then subtracted from the average red pixel, and the difference is 
multiplied by a constant to generate the V component at the center of the 3x3 box, 
V33. The generated V component can then be stored in chrominance output array 58 
or further operated upon for compression or packing. 

[0075] Figures 8A-D illustrate patterns matched for luminance calculation and 

coefficients for generating luminance pixels without intermediate RGB interpolation. 
For the Bayer pattern, any 3x3 box of pixels matches one of four possible patterns. 

[0076] Since the Bayer pattern has twice as many green pixels are either red or blue 

pixels, two of the four patterns have green pixels in the center. Pattern 1 (Fig. 8A) has 
a green center pixel and red at the top and bottom, while pattern 4 (Fig. 8D) has blue 
above and below the green center pixel. The blue pixel is at the center for pattern 2, 
(Fig. 8B), while the red pixel is at the center for pattern 3 (Fig. 8C). 

[0077] Each pixel in a 3x3 pattern is multiplied by a coefficient. The 3x3 matrix of 

coefficients for each pattern is shown below the pattern. For example in Fig. 8A, the 
two red pixels are each multiplied by a coefficient of C/2, while the two blue pixels 
are multiplied by a coefficient of E/2. The center green pixel is multiplied by D/2, 
while the other four green pixels are multiplied by D/8. The average luminance is then 
generated by summing the 9 products. 

[0078] 

The constants C, D, E are chosen to account for variations in perceived intensity of 
the primary red, green, and blue colors and for the color composition of the Bayer 
pattern itself. For example, green is multiplied by constant D which is larger than 
constant E which is multiplied by blue pixels and constant C which is multiplied by red 
pixels. These constants could be adjusted to account for chromatic errors of the light 
sensor, lens, display, or other components. The constant C is 0.299, D is 0.587, and E 
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is 0.144 in this embodiment. 

[0079] Figures 9A-D illustrate the pattern matched for chrominance calculation and 

coefficients for generating intermediate sums without RGB interpolation. Chrominance 
components are only generated for one of the four possible patterns of 3x3 Bayer- 
pattern pixels, pattern 4. Pattern 4 is shown in Fig. 9A, and has a green pixel at the 
center and at the four corners. Blue pixels are at the top and bottom center, with red 
pixels to the right and left of the center green pixel. 

[0080] To generate the average luminance (Y/9) for the 3x3 box, each of the 9 luminance 
pixels generated by the luminance calculator and stored in the 3-line buffer are 
multiplied by 1 /9. Thus the coefficients for generating the Y/9 average are all 1 /9, as 
shown in Fig. 9B. These coefficients are multiplied by the Y pixels rather than the 
Bayer-pattern pixels. 

[0081] To generate the U component, and intermediate U average, S_U, is generated by 
the chrominance calculator. The two blue pixels of pattern 4 (above and below the 
center pixel) are each multiplied by 1 /2, while the other green and red pixels are 
ignored, or multiplied by 0. The coefficients for generating the intermediate U average 
S_U are shown in Fig. 9C. These coefficients are multiplied by the Bayer-pattern pixels 
from the 5-line buffer. 

[0082] To generate the V component, and intermediate V average, S_V, is generated by 
the chrominance calculator. The two red pixels of pattern 4 (to the right and left of 
the center pixel) are each multiplied by 1 /2, while the other green and blue pixels are 
ignored, or multiplied by 0. The coefficients for generating the intermediate V average 
S_V are shown in Fig. 9D. These coefficients are also multiplied by the Bayer-pattern 
pixels from the 5-line buffer. 

[0083] 

To compute the final U and V pixels, the Y/9 luminance average is subtracted 
from the intermediate U and V averages, and then the differences are multiplied by 
different constants. As shown in Fig. 9E, the S_U-Y/9 difference is multiplied by the 
constant 0.493 to generate the U pixel at the center of the 3x3 box. To generate the V 
pixel, the S_V minus Y/9 difference is multiplied by the constant 0.877. The V 
constant is larger than the U constant because the human eye is more sensitive to 
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blue than to red, so red is intensified relative to blue. 

[0084] Various parallel processing techniques may be used that perform the basic 
operations described here in a serial fashion for easier understanding. Figure 10 
shows using an associative array processor to directly generate YUV pixels from an 
RGB Bayer-pattern without intermediate interpolation. Associative processor array can 
use a content-addressable memory array and other logic to perform logic and 
mathematic operations on many data values at once. Patterns of data values (pixels) 
can be searched for and matched and replaced with a result pattern. For example, all 
G pixel values equal to 0.5002 could be searched for and replaced with a Y pixel value 
of 0.2387 or some other value. 

[0085] Figure 1 1 shows using a digital-signal processor (DSP) to perform luminance and 
chrominance calculations. DSP 74 can be programmed to read R, G, and B pixels in 
3x3 boxes from an input Bayer pattern, and perform calculation steps to generate 
luminance and chrominance values. These calculations can be performed in series or 
in parallel when multiple calculation engines are included in DSP 74. 

[0086] Figures 12A-D illustrate for an alternate Bayer pattern initializations the patterns 

matched for chrominance calculation and coefficients for generating intermediate 

D 

ffn sums without RGB interpolation. Some image sensors may initialize the Bayer pattern 

at a different location than shown in Fig. 2. For example, the pattern may begin with a 
red pixel rather than a green pixel, and the entire pattern may then be shifted over by 
one column. Then if U, V generation is still performed at locations (1,1), (1 ,3), (3,1), 
(3,3), etc, the 3x3 box matches pattern 3 (Fig. 8C) rather than pattern 4. 

[0087] Chrominance components are only generated for one of the four possible patterns 
of 3x3 Bayer-pattern pixels, pattern 3. Pattern 3 is shown in Fig. 1 2A, and has a red 
pixel at the center. Green pixels are above, below, to the right and left of the center 
red pixel. Blue pixels are at the four comers. 



s 



[0088] 



To generate the average luminance (Y/9) for the 3x3 box, each of the 9 luminance 
pixels generated by the luminance calculator and stored in the 3-line buffer are 
multiplied by 1 /9 as described before. The coefficients for generating the Y/9 average 
are all 1 /9, as shown in Fig. 1 2B. These coefficients are multiplied by the Y pixels 
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rather than the Bayer-pattern pixels. 

[0089] To generate the U component, and intermediate U average, S_U, is generated by 
the chrominance calculator. The four blue pixels of pattern 3 (above and below and to 
the left and right of the center pixel) are each multiplied by 1 /4, while the other green 
and red pixels are ignored, or multiplied by 0. The coefficients for generating the 
intermediate U average S_U are shown in Fig. 1 2C. These coefficients are multiplied by 
the Bayer-pattern pixels from the 5-line buffer. 

[0090] To generate the V component, and intermediate V average, S_V, is generated by 
the chrominance calculator. The one red pixels of pattern 3 (the center pixel) is 
multiplied by 1 , while the other green and blue pixels are ignored, or multiplied by 0. 
The coefficients for generating the intermediate V average S_V are shown in Fig. 1 2D. 
These coefficients are also multiplied by the Bayer-pattern pixels from the 5-line 
buffer. 



SI 



[0091] To compute the final U and V pixels, the Y/9 luminance average is subtracted 
from the intermediate U and V averages, and then the differences are multiplied by 
different constants. As shown in Fig. 12E, the S.U-Y/9 difference is multiplied by the 
constant 0.493 to generate the U pixel at the center of the 3x3 box. To generate the V 
y pixel, the S_V minus Y/9 difference is multiplied by the constant 0.877. The V 

constant is larger than the U constant because the human eye is more sensitive to 
JSj blue than to red, so red is intensified relative to blue. 

rSr. 
jj 

[0092] Similar alterations can be made for other initializations of the Bayer pattern, such 
as when patterns 1 or 2 occur around pixel location 3,3. When chrominance- 
generating locations occur at other locations, then these locations can be matched to 
one of the four patterns. For example, chrominance may be generated at locations 
(2,2), (2,4), (4,2), (4,4), etc. 

[0093] ALTERNATE EMBODIMENTS 

[0094] Several other embodiments are contemplated by the inventors. For example the 
white balancer could be removed. Other modules can be added such as for edge 
enhancement and color enhancement. Edge enhancement can be performed on the 
luminance while' color enhancement could be performed on the chrominance 



APP ID=10064177 



Page 15 



# 



components only, for example. Other kinds of image sensors could be substituted, 
and additional buffering and pipeline registers can be added at several points in the 
data path. Parallel data paths could be used to increase throughput. Larger buffers 
could be used, such as a 7-line Bayer-pattern buffer or a 9-line luminance buffer, or 
even full-frame buffers. Other YUV formats could be substituted, such as YUV 4:2:0 or 
YUV 4:2:2 format. Even YUV 4:4:4 format could be used, although with less efficiency. 
Pixels in the five-line or 3-line buffer do not have to physically be stored in the 
arrangement shown. For large memories, one physical memory row could store all five 
rows of pixels. Various interleaving and mapping schemes could alter the actual 
storage locations to optimize bandwidth or other design parameters. Many memory 
arrangements, both physical and logical, are possible. 

[0095] The 3-line buffer could be loaded directly from the luminance calculator, or the 
luminance calculator could load the output Y array or other storage, and then the 3- 
line buffer be loaded from this output Y array. The chrominance calculator could also 
directly read the output Y array or other memory. Various parallel processing 
techniques may be used that perform the basic operations described here in a serial 
fashion for easier understanding. 

[0096] Rather than use the pattern 4 shown in Fig. 9A for chrominance calculations, other 
patterns could be substituted, such as pattern 1 . Appropriate input and coefficient 
changes can be made by a person of skill in the art. Partial calculations may be 
performed near the edges of the Bayer pattern, such as for pixel locations 1 ,3 and 1,1 
and 3,1 . These edge locations lack the normal number of input pixels and must be 
averaged over a smaller number of pixel inputs. 

[0097] A small temporary buffer can be used for the 3- and 5-line buffers. These buffers 
can be merged or expanded, and can receive either the newest pixels or the oldest 
pixels in the line. The size of the line buffer can also be varied for different horizontal 
resolutions. 

[0098] Different segmentation of the pixel pipeline can be substituted, depending on 
logic delays in each pipestage. Rapid* clocking and slow adders may require 2 clock 
cycles for each adder or multiplier in the calculators, or slower clocks and faster logic 
may allow both horizontal and vertical adders to operate in a single pipestage without 
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actually latching in the intermediate values. Similar adjustments and modifications can 
be made to other stages and functional units. 

[0099] Different data encodings and primary colors can be used. Bit widths can be varied. 
Many data formats may be used with the invention. Additional functions can be added. 
Many arrangements of adders, shifters, and logic units are possible. Adders may be 
reused or used recursively. Some image sensors may alter the Bayer pattern in 
different ways, such as by producing data from lower lines before upper lines. Various 
modifications can be made as needed to accommodate these changes. 

The abstract of the disclosure is provided to comply with the rules requiring an 
abstract, which will allow a searcher to quickly ascertain the subject matter of the 
technical disclosure of any patent issued from this disclosure. It is submitted with the 
understanding that it will not be used to interpret or limit the scope or meaning of the 
claims. 37 C.F.R. § 1 .72(b). Any advantages and benefits described may not apply to 
all embodiments of the invention. When the word "means" is recited in a claim 
element, Applicant intends for the claim element to fall under 35 USC § 1 1 2, 
paragraph 6. Often a label of one or more words precedes the word "means". The 
word or words preceding the word "means" is a label intended to ease referencing of 
claims elements and is not intended to convey a structural limitation. Such means- 
plus-function claims are intended to cover not only the structures described herein 
for performing the function and their structural equivalents, but also equivalent 
structures. For example, although a nail and a screw have different structures, they 
are equivalent structures since they both perform the function of fastening. Claims 
that do not use the word means are not intended to fall under 35 USC § 1 1 2, 
paragraph 6. Signals are typically electronic signals, but may be optical signals such 
as can be carried over a fiber optic line. 

[0101] The foregoing description of the embodiments of the invention has been 

presented for the purposes of illustration and description. It is not intended to be 
exhaustive or to limit the invention to the precise form disclosed. Many modifications 
and variations are possible in light of the above teaching. It is intended that the scope 
of the invention be limited not by this detailed description, but rather by the claims 
appended hereto. 
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[0100] 
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